System and method for generating a product recommendation in a virtual try-on session

ABSTRACT

System and methods for generating a product recommendation based on reactions such as bio-feedback of a subject during a virtual try-on session are described herein. A recommendation engine captures bio-feedback and determines whether the subject has a positive or negative attitude towards a certain feature of a product that is being virtually “tried on” with the subject. The recommendation engine can then provide a product recommendation based on the actual sentiment of the subject towards a product feature.

BACKGROUND

The present disclosure is directed to product recommendations, and moreparticularly to generating product recommendations based on bio-feedbackfrom a subject during a virtual try-on session.

SUMMARY

Existing recommendation engines provide product recommendations based onpreferences derived from deliberate and intended user inputs such as,for example, search terms, expressed preferences, purchases, productviewings, or the like. Recommendations generated from such systems areoften inaccurate because such systems cannot capture user sentimentsassociated with a product, or its particular features, that are not partof such inputs. For example, a user may enter the search query “greenapron”. After seeing a search result (e.g., an image), the user may likethe product but not its particular shade of green. Currentrecommendation engines cannot capture the user's reaction and learnabout the preference until it is reflected in the user's inputs.

To solve the problem and provide more accurate recommendations, systemsand methods are disclosed herein for generating product recommendationsbased on bio-feedback of a subject captured during a virtual try-onsession. As described below, a recommendation engine is implemented atuser equipment or a server to generate a recommendation based onbio-feedback captured from a subject during the subject's interactionwith the virtual try-on session. The virtual try-on session is asimulated visualization illustrating the subject trying on a product.

The recommendation engine captures bio-feedback such as the subject'sfocal point, the subject's line of sight, facial expressions, verbalexpressions, gestures, movements, biometric measurements, and/or thelike of the subject, to determine what feature of the product the useris paying attention to, and whether the user likes, or dislikes theparticular feature of the product.

In some embodiments, the bio-feedback includes biometric measurementssuch as the pulse rate, blood pressure, pupil dilation, and/or the like.The recommendation engine determines a change of a biometric parameterfrom the biometric measurement and queries a biometric database based onthe change of the biometric parameter to identify the emotion that isimplied by the change of the biometric parameter.

In some embodiments, the bio-feedback includes a facial expression, agesture or a body movement of the subject, and/or the like. Therecommendation engine captures an image or video of a subject's movement(including facial movement) and generates a movement pattern or facialexpression pattern from the captured image or video content. Therecommendation engine then uses the pattern to identify the movement orfacial expression, and then identifies an emotion associated with theidentified movement.

Based on the particular feature and the identified emotion, therecommendation engine recommends a product having the same particularproduct feature if the bio-feedback shows positive emotion, or avoidsrecommending products having the same feature if the bio-feedback showsnegative emotion. In this way, the recommendation engine improvescomputer-assisted shopping experience by providing more accurate andmore efficient product recommendations that captures the actualsentiment of a subject towards a particular product feature during avirtual try-on experience.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects and advantages of the disclosure will beapparent upon consideration of the following detailed description, takenin conjunction with the accompanying drawings, in which like referencecharacters refer to like parts throughout, and in which:

FIG. 1 depicts an illustrative embodiment illustrating aspects of asubject interacting with a simulated visualization of a virtual try-onof a product, according to some embodiments described herein;

FIG. 2 depicts an illustrative embodiment illustrating aspects ofproviding a product recommendation based on bio-feedback captured fromsubject interactions with the simulated visualization described in FIG.1 , according to some embodiments described herein;

FIG. 3 depicts an example block diagram illustrating the exchange ofdata artifacts between a simulation engine, the product and variousdatabases to generate a product recommendation as described in FIG. 2 ,according to some embodiments described herein;

FIG. 4 depicts an illustrative flowchart of a process for generating aproduct recommendation based on bio-feedback during a virtual try-onsession, in accordance with some embodiments of the disclosure;

FIG. 5 depicts an illustrative flowchart of a process for generating asimulated visualization of a virtual try-on of a product, in accordancewith some embodiments of the disclosure;

FIG. 6 depicts an illustrative flowchart of a process for detecting aninteraction with the simulated visualization illustrating a productfeature of the product, in accordance with some embodiments of thedisclosure; and

FIG. 7 depicts an illustrative flowchart of a process for capturingbio-feedback and determining an emotion indicator from the bio-feedback,in accordance with some embodiments of the disclosure.

DETAILED DESCRIPTION

FIG. 1 depicts an illustrative embodiment illustrating aspects of asubject interacting with a simulated visualization of a virtual try-onof a product, according to some embodiments described herein. Diagram100 shows a subject 102, which is a human user in this example,operating user equipment 114, on which a simulated visualization 112 ofa virtual try-on of a product, e.g., lipstick product “Revlon Coral”113, is being presented. The user equipment 114 may be a personalcomputer (PC), a laptop computer, a tablet computer, a hand-heldcomputer, a stationary telephone, a personal digital assistant (PDA), amobile telephone, a smart phone, an intelligent wearable device, or anyother smart equipment, computing equipment, or wireless device, and/orcombination of the same. In some embodiments, the user equipment 114 mayhave a front facing screen and a rear facing screen, multiple frontscreens, or multiple angled screens. In some embodiments, the userequipment 114 may have a front facing camera and/or a rear facingcamera. In some embodiments, the user equipment 114 may have an audiorecorder. In some embodiments, the user equipment 114 may have varioussensors including a gyroscope and/or an accelerometer. In someembodiments, the user equipment 114 may be equipped with, or may receivedata from a wearable device that is capable of biometric measurementincluding measuring the pulse rate, heart rate, body temperature, bloodpressure, and/or the like.

An enlarged view of the screen of user equipment 114 is shown at 112,which illustrates a simulated visualization of a human headshot. The liparea 116 of the human face has been accentuated with a lipstick colorcorresponding to the lipstick product 113, creating a virtual experiencethat the lipstick product “Revlon Coral” 113 is being “tried” by asubject relating to the human headshot.

Specifically, a recommendation engine, implemented at user equipment114, may generate the simulated visualization 112 to illustrate a visualtry-on experience. For example, the subject 102 may select the lipstickproduct 113 and submit an image or a video of the subject (e.g., animage of the subject 102 herself or another human user, etc.) for thelipstick product 113 to be tried on.

In some embodiments, the recommendation engine may determine a type ofthe product, e.g., beauty products for the lipstick 113, from which therecommendation engine determines that an image or video showing thefacial area of a human body is to be used. The recommendation engine mayoptionally generate for display, on the user equipment 114, instructionsfor the subject 102 to capture or submit a photo showing the facial areaof a human body. Or alternatively, the recommendation engine mayoptionally adjust a submitted photo to focus on the facial area of ahuman body, e.g., by cropping a submitted image showing a full humanbody of the subject 102 to concentrate on the facial area of the subject102.

The recommendation engine may then identify the product feature of thelipstick product 113, e.g., the lipstick color, from a product database(e.g., see 219 described in FIG. 3 ). The recommendation engine thenidentifies the lip area 116 of the human face in the submitted (oradjusted) image or video, and change the original color of the lip area116 to the lipstick color “coral” corresponding to the lipstick product“Revlon Coral” 113. In this way, the recommendation engine generates asimulated visualization 112 that shows a human face having the lip area116 virtually painted “coral” by the lipstick product 113.

In some embodiments, the subject 102 may submit a video of the subject(e.g., a human body) to generate a dynamic view of the virtual try-on.The recommendation engine may generate a number of video frames from thevideo of the subject. For each video frame showing the facial area ofthe human body, the recommendation engine may modify the color of thelip area to the lipstick color as described above. In this way, therecommendation engine may generate a video simulating a dynamic view ofthe human face having the lip area 116 virtually painted by the lipstickproduct 113.

In some embodiments, instead of retrieving the product features from adatabase and generating the simulated visualization 112, therecommendation engine at user equipment 114 may receive, from anotherapplication on the user equipment, another user equipment, a remote datasource, and/or the like, the simulated visualization 112 in the form ofa composite image or video content for display via a user interface. Inthis case, the recommendation engine may retrieve a content structureassociated with the simulated visualization 112, which includesstructural data attributes. For example, the structural data attributesinclude a data structure for the lip area and attributes indicative ofthe current color “coral” of the lip area 116. The recommendation enginemay then determine whether the color attribute of the lip area 116 hasbeen modified based on metadata associated with the simulatedvisualization 112. In this way, if a modification of the color attributeis shown in the modification history of the simulated content, therecommendation engine may identify that the current color “coral” of thelip area 116 relates to a product feature imposed on the original imagefor virtual try-on experience.

Upon identifying the product feature of “coral” color, therecommendation engine may detect an interaction from subject 102 withthe simulated visualization 112. As referred to herein, the term“interaction” refers to an action of a subject intended by the subjectto be an express input to the system implementing the recommendationengine. For example, subject 102 may operate the user equipment 114 tomove to the center of the screen, or enlarge a portion of the simulatedvisualization 112 so that the subject 102 can concentrate on the liparea 116. For another example, subject 102 may modify the simulatedvisualization 112 by change a color tone, contrast, brightness,saturation, and/or the like of the simulated visualization 112.

The recommendation engine may capture bio-feedback from the subject 102.As referred to herein, the term “bio-feedback” refers to sensedbiological functions of the subject from which the subject's attentionor sentiment may be identified. Such biological functions are sensed bya system using appropriate sensors and are not provided by the subjectto the system as an intended and deliberate input to the system. Forexample, bio-feedback can be any of a focal point, a line of sight,facial expression, an utterance, a body movement, a gesture, a pose,biometrics (e.g., blood pressure, pulse rate, heart rate, pupildilation, electroencephalogram (EEG), body temperature, and/or thelike), and/or the like.

For example, the recommendation engine may engage a camera on the userequipment 114 to detect that a focal point or a line of sight 119 of thesubject 102 is directed to the lip area 116 in the simulatedvisualization 112. For another example, the recommendation engine mayengage a camera with the user equipment 114 to capture an image or videoof the subject 102 showing a facial expression, a movement, a gesture,an utterance of the subject 102. For another example, the recommendationengine may engage an audio recorder at user equipment 114 to record anaudio clip of the subject making an utterance. For another example, therecommendation engine may engage a gyroscope sensor and/or anaccelerometer at the user equipment 114 to capture a movement of thesubject 102. For another example, the recommendation engine may obtainbiometrics measurement from a device (e.g., a wristband, a headband, orany wearable device, etc.) in contact with the subject 102, such as apulse rate, a heart rate, a blood pressure, body temperature,electroencephalogram (EEG), and/or the like.

As shown in diagram 100, the subject 102 may contemplate (with orwithout a verbal expression) “this color looks great on me” 117, and themental state may be reflected in a smiling facial expression. Therecommendation engine may capture the changed facial expression, anddetermines the facial expression as “smile,” which exhibits a positiveattitude from the subject 102. Thus, the recommendation engine maycorrelate the gaze 119 directed to the lip area 116 being painted withthe lip color “coral” with the positive sentiment shown from the facialexpression of the subject 102. The recommendation engine may determinethat the lip color “coral” is favored by subject 102.

FIG. 2 depicts an illustrative embodiment illustrating aspects ofproviding a product recommendation based on bio-feedback captured fromsubject interactions with the simulated visualization described in FIG.1 , according to some embodiments described herein. Diagram 200 shows anexample screen of user equipment 114, which illustrates a productrecommendation 121 of another lipstick product “Vincent Logo lip staincoral” 122 which has a similar color “coral” with the product “Revloncoral” 113 that has been virtually tried on. As diagram 100 illustratesthat the subject 102 exhibits a positive emotion towards the lipstickcolor “coral” during the virtual try-on, the recommendation enginegenerates another product 122 having a similar product feature, e.g.,the color “coral” to the subject 102.

In another example, the subject 102 may interact with the visualization112 by changing the color tone of the visualization 112. Therecommendation engine may capture a “smiling” facial expressionreflecting the positive sentiment 117 after the color tone of thevisualization 112 has been changed, and may then determine that thepositive attitude from the subject 102 is related to the changed colortone, e.g., a lighter lip color. Therefore, the recommendation enginemay provide a product recommendation of another lipstick product havingthe lighter color.

FIG. 3 depicts an example block diagram illustrating the exchange ofdata artifacts between a simulation engine, the product and variousdatabases to generate a product recommendation as described in FIG. 2 ,according to some embodiments described herein. Diagram 300 shows aproduct database 219, a simulation engine 305, a recommendation engine310, and/or other entities interact with each other to exchange variousdata messages, artifacts, and/or the like.

In some embodiments, the product database 219 may be housed at anelectronic storage device located remotely from the simulation engine305 or the recommendation engine 310, and may be accessible via acommunication network. As referred to herein, the phrase “electronicstorage device” or “storage device” should be understood to mean anydevice for storing electronic data, computer software, or firmware, suchas random-access memory, read-only memory, hard drives, optical drives,digital video disc (DVD) recorders, compact disc (CD) recorders, BLU-RAYdisc (BD) recorders, BLU-RAY 3D disc recorders, digital video recorders(DVR, sometimes called a personal video recorder, or PVR), solid statedevices, quantum storage devices, gaming consoles, gaming media, or anyother suitable fixed or removable storage devices, and/or anycombination of the same. The product databases 219 may also be accessedat a cloud-based storage, which may be used to supplement a localstorage device or instead of the storage device. Product information maybe stored at the product database 219 in a structural format, includinga listing of data fields describing attributes of the product such as aproduct ID, product brand name, category, type, make and model, color,shape, ingredients, retail price, and/or the like.

In some embodiments, the simulation engine 305 or the recommendationengine 310 may be implemented at different devices and may communicatewith each other via a communication network. Or alternatively, thesimulation engine 305 or the recommendation engine 310 may be integratedinto one application running on the same device, e.g., user equipment114 in FIG. 1 . Any of the simulation engine 305 or the recommendationengine 310 (referred to jointly, separately or interchangeably as therecommendation engine described throughout the disclosure) may beimplemented using any suitable architecture. For example, any of thesimulation engine 305 or the recommendation engine 310 may be astand-alone application wholly-implemented on user equipment device. Insuch an approach, instructions of the simulation engine 305 or therecommendation engine 310 are stored locally (e.g., at storage of theuser equipment 114 in FIG. 1 but communicate with a remote productdatabases 219 in FIG. 3 ), and data for use by the engine is downloadedon a periodic basis (e.g., from an out-of-band feed, from an Internetresource such as the product database 219 in FIG. 3 , or using anothersuitable approach). Control circuitry may be used to retrieveinstructions of simulation engine 305 or the recommendation engine 310from storage and process the instructions to generate any of thedisplays discussed herein such as the simulated visualization 112displayed at a user interface on the user equipment 114. Based on theprocessed instructions, control circuitry may determine what action toperform when input is received from input interface, e.g., a touchscreen of user equipment 114. For example, movement of a fingertip on atouch screen up/down may be indicated by the processed instructions whenthe input interface indicates that an up/down motion is detected.

In some embodiments, the simulation engine 305 or the recommendationengine 310 is a client-server based application. Data for use by a thickor thin client implemented on user equipment device is retrievedon-demand by issuing requests to a server remote to the user equipmentdevice. In one example of a client-server based simulation engine 305 orthe recommendation engine 310, control circuitry runs a web browser thatinterprets web pages provided by a remote server. For example, theremote server may store the instructions for the application in astorage device.

In some embodiments, the simulation engine 305 or the recommendationengine 310 is downloaded and interpreted or otherwise run by aninterpreter or virtual machine (run by control circuitry). In someembodiments, the simulation engine 305 or the recommendation engine 310may be encoded in the ETV Binary Interchange Format (EBIF), received bycontrol circuitry as part of a suitable feed, and interpreted by a useragent running on control circuitry. For example, the simulation engine305 or the recommendation engine 310 may include an EBIF application. Insome embodiments, the simulation engine 305 or the recommendation engine310 may be defined by a series of JAVA-based files that are received andrun by a local virtual machine or other suitable middleware executed bycontrol circuitry running on user equipment 114. In some of suchembodiments (e.g., those employing MPEG-2 or other digital mediaencoding schemes), the simulation engine 305 or the recommendationengine 310 may be, for example, encoded and transmitted in an MPEG-2object carousel with the MPEG audio and video packets of a program.

The simulation engine 305 or the recommendation engine 310 forperforming any of the embodiments discussed herein may be encoded oncomputer-readable media. Computer-readable media includes any mediacapable of storing data. The computer readable media may be transitory,including, but not limited to, propagating electrical or electromagneticsignals, or may be non-transitory including, but not limited to,volatile and non-volatile computer memory or storage devices such as ahard disk, floppy disk, USB drive, DVD, CD, media cards, registermemory, processor caches, Random Access Memory (“RAM”), etc.

The simulation engine 305 is configured to receive product features 202from the product databases 219, e.g., via a communication network, andan image or video of a subject 201, e.g., via a user interface on userequipment 114. The simulation engine 305 is then configured to generatea simulated visualization, which is presented for display via a userinterface at user equipment, e.g., as shown at 112 in FIG. 1 . Forexample, the simulated visualization 112 includes one or more attributesin the submitted image 201 being modified with the product feature 202.The process of generating a simulated visualization 112 is furtherdescribed in relation to FIG. 5 .

The recommendation engine 310 is configured to monitor any interactionand bio-feedback from a subject 102 when the simulated visualization 112is presented at the user interface on user equipment 114. Interactiveactivities such as a swipe, adding a filter, a zoom-in action directedto a portion of the simulated visualization, and bio-feedback such as anaccelerated pulse rate, a smile, a frown, and/or the like, are capturedby the recommendation engine 310 to determine an emotion indicator ofthe subject 102. The recommendation engine 310 is configured toassociate an interactive activity with at least one form of subsequentor simultaneous bio-feedback to determine an emotion indicator of thesubject 102, as further described in FIGS. 6-7 . When the emotionindicator shows a positive attitude, e.g., via a positive facialexpression, an utterance with a positive tone or words, a gestureshowing satisfaction, etc., the recommendation engine 310 determines theproduct feature 202 that the subject 102 is paying attention to andtriggers the bio-feedback as a preferred feature 215. The recommendationengine 310 may then transmit a query based on the preferred feature 215to the product database 219 for a product recommendation 216. Forexample, as shown in FIG. 1 , the queried product recommendation of“Vincent Logo Lip stain Coral” 122 has the preferred feature of thecolor “coral” determined from the virtual try-on visualization 112 ofthe product “Revlon Coral” 133.

FIGS. 4-7 provide example flowcharts illustrating various processesimplemented by the recommendation engine as discussed throughout thedisclosure, and specifically the simulation engine 305 or therecommendation engine 310 described in FIG. 3 . Processes 400-700 may beexecuted by control circuitry (e.g., control circuitry that isconfigured to control, instruct or implement the simulation engine 305or the recommendation engine 310 described in FIG. 3 ). Controlcircuitry may be part of user equipment 114, or of a remote serverseparated from the user equipment by way of a communications network.Specifically, control circuitry may be based on any suitable processingcircuitry, and comprises control circuits and memory circuits which maybe disposed on a single integrated circuit, or may be discretecomponents. As referred to herein, processing circuitry should beunderstood to mean circuitry based on one or more microprocessors,microcontrollers, digital signal processors, programmable logic devices,field-programmable gate arrays (FPGAs), application-specific integratedcircuits (ASICs), etc., and may include a multi-core processor (e.g.,dual-core, quad-core, hexa-core, or any suitable number of cores) orsupercomputer. In some embodiments, processing circuitry may bedistributed across multiple separate processors or processing units, forexample, multiple of the same type of processing units (e.g., two IntelCore i7 processors) or multiple different processors (e.g., an IntelCore i5 processor and an Intel Core i7 processor). Some control circuitsmay be implemented in hardware, firmware, or software.

FIG. 4 depicts an illustrative flowchart of a process for generating aproduct recommendation based on bio-feedback during a virtual try-onsession, in accordance with some embodiments of the disclosure. Process400 begins at 402, where control circuitry receives, from the subject(e.g., 102 in FIG. 1 ), a selection of a product (e.g., “Revlon Coral”113 in FIG. 1 ) for virtual try-on. At 404, control circuitry receivesan image or video of the subject (e.g., 201 in FIG. 3 ) on which theproduct is to be applied. For example, control circuitry of userequipment 114 causes a camera on the user equipment 114 to capture animage of the subject, e.g., the subject 102 herself. For anotherexample, control circuitry obtains, via a communication interface, animage via a communication message such as but not limited to anelectronic mail, text message, Internet instant message, downloadingfrom an Internet source, and/or the like.

At 406, control circuitry generates a simulated visualization (e.g., 112in FIG. 1 ) illustrating the product being applied on the subject, theprocess of which is further illustrated in FIG. 5 .

At 408, control circuitry identifies a product feature in the simulatedvisualization of the product. For example, control circuitry receivesthe product feature (e.g., 202 in FIG. 3 ) as a structural dataattribute from a product database 219. For another example, controlcircuitry analyzes image content and metadata of the simulatedvisualization to determine whether an attribute of one or more pixelshas been altered. If attributes (e.g., color code) relating to a set ofpixels have been modified, control circuitry determines that themodified attribute such as a changed color is related to a productfeature.

At 410, control circuitry detects an interaction with the simulatedvisualization illustrating a product feature of the product. Forexample, control circuitry receives signals representing a fingertipmovement from sensors on a touch screen of user equipment 114 to detectan interaction from the subject 102, e.g., to move a portion of thesimulated visualization to the center, or to enlarge the portion of thesimulated visualization to the center, and/or the like. Controlcircuitry further receives signals detected by a gyroscope and/or anaccelerometer equipped with the user equipment indicative of a movementof user equipment, e.g., to move or position the user equipment by thesubject to view the displayed simulated visualization at a certainangle, etc. Further detail relating to detecting the interaction withthe simulated visualization is described in FIG. 6 .

At 412, control circuitry captures at least one form of bio-feedbackfrom the subject 102, based on which control circuitry determines anemotion indicator at 414. For another example, control circuitryreceives signals representing image data of the eye area of the subject102, from a camera on user equipment 114. Control circuitry thendetermines, from the image data of the eye area of the subject 102, aline of sight or focal point of the subject to determine which portionof the simulated visualization the subject 102 is paying attention to.Further detail relating to capturing bio-feedback are further describedin FIG. 7 .

At 416, control circuitry determines whether the emotion indicator fromstep 414 is positive. If the emotion indicator is positive, process 400proceed with 418, where control circuitry generates a recommendation foranother product having the same product feature that the subject ispaying attention to. For example, control circuitry transmits a querybased on the product feature (such as the lip color “coral” in FIG. 1 )to the product databases (e.g., 219 in FIG. 3 ), and receives a productrecommendation having the same product feature. If the emotion indicatoris negative, process 400 proceed with 420, where control circuitrystores the at least one product feature as a non-preference for productrecommendations, and restricts a recommendation of another producthaving the non-preferred product feature at 422.

FIG. 5 depicts an illustrative flowchart of a process for generating asimulated visualization of a virtual try-on of a product (406 in FIG. 4), in accordance with some embodiments of the disclosure.

Process 500 begins at 502, where control circuitry determines a type ofthe product. For example, control circuitry obtains product informationfrom the product database 219 in FIG. 3 , which includes a product typeattribute such as beauty, electronics, apparel, house supplies, and/orthe like. At 504, control circuitry determines at least one aspect ofthe subject where the product is to be applied to based on the type ofthe product. For example, control circuitry obtains applicationinformation relating to the product type from the product database,e.g., beauty products are applied to the facial area of a human body,house supplies are applied to an indoor setting, apparels area appliedto a human body, and/or the like. At 506, upon identifying where theproduct is to be applied, control circuitry may optionally generate fordisplay, e.g., at a user interface on user equipment 114, an instructionrequesting a submission of an image illustrative of the at least oneaspect of the subject. For example, for beauty products such as “RevlonCoral” 113 shown in FIG. 1 , control circuitry instructs a front cameraequipment 114 to capture a photo that focuses on the face of subject112.

At 507, control circuitry determines whether the submitted image showsthe aspect of the subject at which the product is to be applied. If thesubmitted image does not focus on the aspect of the subject, e.g., animage of a full human body is received, process 500 moves to 508, wherecontrol circuitry generates an adjusted image from the submitted imageto focus on the at least one aspect. For example, control circuitrycrops the submitted image of the full human body into a head shot thatconcentrates on the facial area. When the submitted image focuses on theaspect of the subject at 507, or after adjustment at 508, process 500moves to 510, where control circuitry determines an attribute of theaspect of the subject relating to the product type. For example, asshown in FIG. 1 , for beauty products such as a lipstick “Revlon Coral”113, control circuitry has identified that the product is to be appliedon the lip area on a human face from step 504, and may then identify,through pattern recognition and edge detection, the lip area (e.g., 116in FIG. 1 ) on the image of the subject. Control circuitry alsodetermines the color attributes, e.g., the red, green, blue (RGB)parameters of pixels belonging to the lip area, which are to be modifiedby color attributes of the product.

At 512, control circuitry identifies a product feature associated withthe product, e.g., received at 202 from a product databases 219. At 514,control circuitry modifies the attribute of the aspect of the productbased on the identified product feature. For example, control circuitrymodifies the RGB parameters of pixels belonging to the lip area with aset of new RGB parameters that reflect the lipstick color of “RevlonCoral” 113 to generate the simulated visualization (e.g., 112 in FIG. 1).

FIG. 6 depicts an illustrative flowchart of a process for detecting aninteraction with the simulated visualization illustrating a productfeature of the product (410 in FIG. 4 ), in accordance with someembodiments of the disclosure.

At 602, control circuitry determines a portion of the simulatedvisualization towards which the interaction from the subject isdirected. For example, control circuitry determines, via signals fromsensors on the touch screen of user equipment 114, that the portion ofthe simulated visualization has been moved to the center of the screen,or has been enlarged for detailed view.

At 606, control circuitry determines whether the portion of thesimulated visualization is illustrative of the product feature. Forexample, control circuitry determines whether the portion that thesubject 102 is paying attention to has been modified by the productfeature. In the example shown in FIG. 1 , the lip area 116 has beenmodified by the product feature, e.g., the lipstick color “coral.” Whenthe portion of the simulated visualization is illustrative of theproduct feature, process 600 proceeds with 608, where control circuitryassociates any bio-feedback (obtained at 412 in FIG. 4 ) with theproduct feature.

At 606, when the portion of the simulated visualization that the subjectis paying attention to is not illustrative of the product feature (e.g.,the line of sight 119 or a zoom-in action may be directed to theforehead area in 112), process 600 moves on to 610, where controlcircuitry determines a product type that is related to the portion ofthe simulated visualization towards which the interaction from thesubject is directed. For example, when the lines of sight 119 isdirected to the forehead area, control circuitry transmits a query,based on the portion “forehead,” to the product database (219 in FIG. 3) for a product relating to the area “forehead.” At 612, controlcircuitry generates a recommendation for a product having the producttype related to the portion that the subject is paying attention to. Forexample, control circuitry receives, in response to the query, and inturn generates a recommendation of a blemish concealer, a contouringstick, a highlighter, and/or the like that is related to the area“forehead.” In this way, by detecting interactions with the simulatedvisualization, control circuitry can assist a subject (102 in FIG. 1 )to virtually try on more products based on the attention and interest ofthe subject.

FIG. 7 depicts an illustrative flowchart of a process for capturingbio-feedback and determining an emotion indicator from the bio-feedback(412-414 in FIG. 4 ), in accordance with some embodiments of thedisclosure.

Process 700 begins at 702, where control circuitry determines types ofbio-feedback that is available. For example, control circuitry at userequipment (114 in FIG. 1 ) queries a local hardware inventory todetermine sensors that are equipped with the user equipment, such as acamera to capture imagery of the subject, an audio recorder to record anutterance of the subject, a gyroscope and/or accelerometer to capturemovement of the subject, and/or the like. For another example, controlcircuitry at user equipment determines whether a wearable device (e.g.,an electronic wrist band, a headset, and/or the like) is paired withuser equipment, and then receives biometric measurement data from thewearable device, such as pulse rate, heart rate, blood pressure, EEGwave, body temperature, and/or the like.

When biometric measurement data is available, process 700 proceeds with704, where control circuitry obtains the biometric measurement from abiometric measurement device such as a wearable device described above,associated with the subject. At 706, control circuitry determines achange of a biometric parameter from the biometric measurement. Forexample, control circuitry determines an increased pulse rate, aheightened blood pressure, increased EEG activities, and/or the like. At708, control circuitry transmits a query based on the change ofbiometric measurement, to a biometric data table. The biometric datatable may be stored locally at user equipment (114 in FIG. 1 ), or maybe stored at a cloud storage that is remote to the user equipment butaccessible via a communication network. The biometric data table storesbiometric data and related emotion indicators related to the biometricdata. For example, a data entry in the biometric data table may specifya range of pulse rate and an associated emotional state as “excitement.”

At 710, in response to the query, control circuitry obtains an emotionindicator related to the change of the biometric parameter. For example,when the changed pulse rate falls within a range of pulse rate specifiedin the biometric data table, the query result reflects the correspondingemotional state from the data table.

At 702, when no biometric measurement is available, process 700 moves to712, where control circuitry captures content representing the movementor a facial expression of the subject. For example, the content may bevideo content captured by a camera on the user equipment. For anotherexample, the content may be a movement pattern captured by a gyroscopeand/or accelerometer on the user equipment. At 704, control circuitrygenerates a movement pattern of the subject from the captured content.For example, when the content includes video content of a facialexpression of the subject, control circuitry performs patternrecognition on the captured video or image content to identify theposition of eyes, the lip and nose of the facial area, and generate afacial expression. For another example, control circuitry obtains amovement trajectory or position change detected by the accelerometerand/or gyroscope.

At 716, control circuitry transmits a query based on the movementpattern, to a movement database to identify the movement. The movementdatabase may be stored locally at user equipment (114 in FIG. 1 ), ormay be stored at a cloud storage that is remote to the user equipmentbut accessible via a communication network. The movement database storesa movement pattern and related emotion indicators related to themovement pattern. For example, a facial expression pattern having anupwardly arched lip and wrinkled nose is identified as a “smile,” whichis related to emotion indicators “happy,” “satisfied,” and/or the likefrom the movement database. For another example, a movement trajectorycontaining a suddenly elevated height is identified as a “jump,” whichis related to emotion indicators “excited,” “joyful” from the movementdatabase.

Process 700 may also optionally move to 718, where control circuitrydetermines whether audio content is available from the content capturedat 712. If no audio content is available, process 700 proceeds to 730.If audio content is available, process 700 proceeds to 720, wherecontrol circuitry determines whether the audio content includes a verbalexpression from the subject, e.g., by speech detection. At 722, when noverbal expression is detected, control circuitry generates a tonepattern from the vocal expression. At 724, when verbal expression isdetected, control circuitry identifies words from the verbal expressionvia speech detection, and may also optionally generate a tone patternfrom the vocal expression at 722. Process 700 proceeds from 722 and 724to 726, where control circuitry queries an emotion database based on theidentified words and tone pattern. The emotion database may be storedlocally at user equipment (114 in FIG. 1 ), or may be stored at a cloudstorage that is remote to the user equipment but accessible via acommunication network. The emotion database stores verbal expressionssuch as words, phrases, idioms, tone patterns, and/or the like and thecorresponding emotion indicators. For example, a verbal expression“looks great” with an acclaiming tone (e.g., 117 in FIG. 1 ) may bemapped to emotion indicators “happy,” “excited” from the emotiondatabase.

At 730, in response to the queries, control circuitry obtains theemotion indicator associated with the bio-feedback. In some embodiments,control circuitry collects and aggregates one or more different forms ofbio-feedback such as facial expression, verbal expression and biometricmeasurement to identify an emotion indicator. For example, steps 704-710and steps 712-726 may be implemented simultaneously or in parallel toaggregate the identified emotion indicators at 730 from different formof bio-feedback. In some embodiments, control circuitry may prioritizeone form of bio-feedback over another to identify an emotion indicator.For example, when a camera is available on user equipment, controlcircuitry prioritizes a captured facial expression as the primary sourceto determine the emotion indicator. For another example, when a verbalexpression is captured by user equipment, control circuitry prioritizesthe captured verbal expression as the primary source to determine theemotion indicator.

It is contemplated that the actions or descriptions of each of FIGS. 4-7may be used with any other embodiment of this disclosure. In addition,the actions and descriptions described in relation to FIGS. 4-7 may bedone in alternative orders or in parallel to further the purposes ofthis disclosure. For example, each of these steps may be performed inany order or in parallel or substantially simultaneously to reduce lagor increase the speed of the system or method. Furthermore, it should benoted that any of the devices or equipment discussed in relation toFIGS. 1-3 could be used to perform one or more of the actions in FIGS.4-7 .

It will be apparent to those of ordinary skill in the art that methodsinvolved in the present disclosure may be embodied in a computer programproduct that includes a computer-usable and/or readable medium. Forexample, such a computer-usable medium may consist of a read-only memorydevice, such as a CD-ROM disk or conventional ROM device, or arandom-access memory, such as a hard drive device or a computerdiskette, having a computer-readable program code stored thereon. Itshould also be understood that methods, techniques, and processesinvolved in the present disclosure may be executed using processingcircuitry. The processing circuitry, for instance, may be ageneral-purpose processor, a customized integrated circuit (e.g., anASIC), or a field-programmable gate array (FPGA) within the contentconstruction engine or the media destruction engine described throughthe disclosure. Furthermore, processing circuitry, or a computerprogram, may update configuration data of the recommendation engine,which may be stored at a storage within the user equipment 114.

The processes discussed above are intended to be illustrative and notlimiting. One skilled in the art would appreciate that the steps of theprocesses discussed herein may be omitted, modified, combined, and/orrearranged, and any additional steps may be performed without departingfrom the scope of the invention. More generally, the above disclosure ismeant to be exemplary and not limiting. Only the claims that follow aremeant to set bounds as to what the present invention includes.Furthermore, it should be noted that the features and limitationsdescribed in any one embodiment may be applied to any other embodimentherein, and flowcharts or examples relating to one embodiment may becombined with any other embodiment in a suitable manner, done indifferent orders, or done in parallel. In addition, the systems andmethods described herein may be performed in real time. It should alsobe noted, the systems and/or methods described above may be applied to,or used in accordance with, other systems and/or methods.

While some portions of this disclosure may make reference to“convention,” any such reference is merely for the purpose of providingcontext to the invention(s) of the instant disclosure, and does not formany admission as to what constitutes the state of the art.

1-50. (canceled)
 51. A method comprising: receiving video depicting anobject and a selection of a first product; generating a modified objectby applying the first product to the object; causing to be presented ata user device an interactive visualization of the modified object;during presentation of the interactive visualization: receiving, via theuser device, an interaction with the interactive visualization thatchanges a product attribute of the first product exhibited by themodified object in the interactive visualization; capturing a pluralityof forms of bio-feedback data associated with the interactivevisualization; selecting, based on the interaction, a form of theplurality of forms of bio-feedback data that is associated with theinteraction and the interactive visualization; determining, based on theselected form of bio-feedback data, an emotion toward the first productand the product attribute; determining whether the emotion is positiveor negative toward the first product and the product attribute; andcausing to be displayed at the user device a product recommendationcomprising a second product that is selected based on the emotion beingpositive or negative towards the first product and the productattribute, wherein the product recommendation is displayed partiallyover the interactive visualization.
 52. The method of claim 51, whereinthe user device is a smart wearable device.
 53. The method of claim 51,wherein the product attribute is a first product attribute, the methodfurther comprising: in response to determining that the emotion isnegative, determining that the first product attribute is anon-preferred attribute; and selecting the second product to have asecond product attribute different from the first product attribute. 54.The method of claim 53, further comprising: restricting recommendationof a product comprising the non-preferred attribute.
 55. The method ofclaim 51, further comprising: determining a product type of the firstproduct; and storing the changed product attribute as a preferredattribute for the product type.
 56. The method of claim 51, furthercomprising: determining a high priority form of bio-feedback data; andwherein the selected form is selected from the plurality of forms ofbio-feedback data based on the high priority form of bio-feedback data.57. The method of claim 51, wherein generating the modified objectcomprises: determining, based on the video depicting the object, aplurality of video frames showing a portion of the object where thefirst product is to be applied; for each video frame of the plurality ofvideo frames: determining pixels of the video frame that belong to theportion of the object; and generating a modified video frame bymodifying the pixels such that the product is applied to the portion;and generating a composite video comprising the modified video frames,wherein the composite video displays the modified object.
 58. The methodof claim 51, further comprising: retrieving a data structure associatedwith the interactive visualization, wherein the data structure comprisesstructural data attributes of the object and the first product; anddetermining a structural data attribute of the data structure that hasbeen modified based on metadata associated with the interactivevisualization.
 59. The method of claim 51, wherein the plurality offorms of bio-feedback data comprises at least one of a biometricmeasurement and a body movement.
 60. The method of claim 51, whereincapturing the plurality of forms of bio-feedback data comprises engaginga plurality of sensors for capturing the plurality of forms ofbio-feedback data.
 61. A system comprising: a user interface configuredto capture an interaction with an interactive visualization; and controlcircuitry configured to: receive video depicting an object and aselection of a first product; generate a modified object by applying thefirst product to the object; cause to be presented at a user deviceassociated with the user interface an interactive visualization of themodified object; during presentation of the interactive visualization:receive, via the user interface, an interaction with the interactivevisualization that changes a product attribute of the first productexhibited by the modified object in the interactive visualization;capture a plurality of forms of bio-feedback data associated with theinteractive visualization; select, based on the interaction, a form ofthe plurality of forms of bio-feedback data that is associated with theinteraction and the interactive visualization; determine, based on theselected form of bio-feedback data, an emotion toward the first productand the product attribute; determine whether the emotion is positive ornegative toward the first product and the product attribute; and causeto be displayed at the user device a product recommendation comprising asecond product that is selected based on the emotion being positive ornegative towards the first product and the product attribute, whereinthe product recommendation is displayed partially over the interactivevisualization.
 62. The system of claim 61, wherein the user device is asmart wearable device.
 63. The system of claim 61, wherein the productattribute is a first product attribute, wherein the control circuitry isfurther configured to: in response to determining that the emotion isnegative, determine that the first product attribute is a non-preferredattribute; and select the second product to have a second productattribute different from the first product attribute.
 64. The system ofclaim 63, wherein the control circuitry is further configured to:restrict recommendation of a product comprising the non-preferredattribute.
 65. The system of claim 61, wherein the control circuitry isfurther configured to: determine a product type of the first product;and store the changed product attribute as a preferred attribute for theproduct type.
 66. The system of claim 61, wherein the control circuitryis further configured to: determine a high priority form of bio-feedbackdata; and wherein the control circuitry is configured to select theselected form from the plurality of forms of bio-feedback data based onthe high priority form of bio-feedback data.
 67. The system of claim 61,wherein the control circuitry, when generating the modified object, isconfigured to: determine, based on the video depicting the object, aplurality of video frames showing a portion of the object where thefirst product is to be applied; for each video frame of the plurality ofvideo frames: determine pixels of the video frame belonging to theportion of the object; and generate a modified video frame by modifyingthe pixels such that the product is applied to the portion; and generatea composite video comprising the modified video frames, wherein thecomposite video displays the modified object.
 68. The system of claim61, wherein the control circuitry is further configured to: retrieve adata structure associated with the interactive visualization, whereinthe data structure comprises structural data attributes of the objectand the first product; and determine a structural data attribute of thedata structure that has been modified based on metadata associated withthe interactive visualization.
 69. The system of claim 61, wherein theplurality of forms of bio-feedback data comprises at least one of abiometric measurement and a body movement.
 70. The system of claim 61,wherein the control circuitry, when capturing the plurality of forms ofbio-feedback data, is configured to engage a plurality of sensors forcapturing the plurality of forms of bio-feedback data.